systems and processes for digital video stabilization through constraint-based rotation leveling

专利摘要:
Summary "Restriction-based rotational leveling digital video stabilization systems and processes" are provided for Restriction-based refining digital video stabilization systems and processes. Digital video data includes a set of picture frames having associated time periods, and a set of camera orientation data having associated time periods may be provided. a refined set of camera orientation data can be generated by minimizing a rotational speed between successive image frames while minimizing the amount of blank regions in a resulting set of refined image frames, reoriented based on the set. refined camera orientation data.
公开号:BR112015013565A2
申请号:R112015013565
申请日:2013-12-11
公开日:2019-12-17
发明作者:Karpenko Alexandre
申请人:Facebook Inc；
IPC主号:

专利说明:

SYSTEMS AND PROCESSES FOR DIGITAL VIDEO STABILIZATION THROUGH ROTATION LEVELING BASED ON RESTRICTION
TECHNICAL FIELD [001] The object of the present invention relates to signal processing. More particularly, the present invention relates to systems and processes for image and video processing.
BACKGROUND [002] Digital still cameras, capable of capturing video, have recently been widely disseminated. Although the resolution and image qualities of these consumer devices have been improved to the point where rival digital single-lens reflex cameras (DSLRs) are, in some scenarios, their video qualities may be significantly worse than those of film cameras. The reason for this deviation in quality can be twofold. First, compared to film cameras, cell phones can be significantly lighter. Therefore, portable video capture on these devices can exhibit a very high degree of camera shake. Second, cell phone cameras may have sensors that use a rolling shutter (RS). On an RS camera, each image row can be exposed to a slightly different time, which, combined with the camera's undamped movement, can result in a flicker in the output video.
[003] Video stabilization is a family of techniques used to reduce frame-by-frame instability, produced by shaking the video camera. In professional cameras, mechanical image stabilization systems (MIS) are commonly used. For example, in an MIS system, the operator can use a harness, which separates the movement of the camera from the movement of the operator's body. Other MIS systems stabilize the camera's optical system instead of the color itself
Petition 870160074035, of 12/08/2016, p. 6/66
2/54 po of the camera. These systems can move the lens or sensor to compensate for small pitch and yaw movements. These techniques work in real time and do not require computation on the camera. However, they are not suitable for mobile devices and inexpensive cameras, due to their price and size. Digital video stabilization systems can be sensitive to noise (for example, fast moving background objects) and require different aspects for tracking. As a result, digital stabilization, based on aspect tracking, is often disrupted, especially in adverse lighting conditions and excessive background movement. In addition, the visual indications of extraction and comparison by the tables can be computationally expensive. What's more, costs increase with video resolution. In some cases, it can be very expensive to perform real-time video stabilization. Consequently, these approaches are rarely used in today's digital cameras. Instead, manufacturers can opt for more robust (and expensive) mechanical stabilization solutions for high-end DSLRs.
SUMMARY OF THE INVENTION [004] The embodiments according to the invention are, in particular, described in the appended claims directed to a process, a storage medium and a system, in which any feature mentioned in a claim category, for example , a process, can also be claimed in another claim category, for example, a system.
[005] To stabilize digital video, computer-readable processes, systems and media implemented on a computer, in one embodiment, they can provide digital video data, including a set of image frames, having associated time periods, and a set of data camera orientation, having associated time periods. A refined set of camera orientation data can be generated by minimizing a rate of rotation between your subjects.
Petition 870160074035, of 12/08/2016, p. 7/66
3/54 cessive image frames, while minimizing an amount of empty regions in a refined set of resulting image frames, reoriented based on the refined set of camera orientation data.
[006] In one embodiment, the amount of empty regions in the resulting refined set of image frames can be minimized to zero.
[007] In one embodiment, the amount of empty regions in the resulting refined set of image frames can be minimized below a threshold value.
[008] In one embodiment, the set of image frames can be distorted, based on the associated time periods for the set of image frames and the refined set of camera orientation data, to form a set of image frames .
[009] In one embodiment, the distortion of the image frame set, based on the associated time periods for the image frame set and the refined camera orientation data set, to form a corrected image frame set , may include splitting an individual picture frame into several subsections. Each subsection can have an associated time frame and camera orientation. Camera distortion, based on the associated time periods for the image frame set and the refined camera orientation data set, to form a corrected image frame set, may include realigning each subsection, based on the periods associated time and camera orientation to form a corrected individual image frame.
[010] In one embodiment, the set of corrected image frames can be displayed as a video.
[011] In one embodiment, the amount of empty regions in the set of refined picture frames can be minimized below a threshold value.
Petition 870160074035, of 12/08/2016, p. 8/66
4/54 [012] In one embodiment, the number of empty regions below the threshold value can be reconstructed.
[013] In one embodiment, the camera orientation data set, having associated time periods, can be provided by a gyroscope on a portable device.
[014] In one embodiment, the set of picture frames can be provided from a camera on a portable device.
[015] In one embodiment, cameras, having associated time periods, can be provided from a gyroscope on a portable device, including a mobile phone and a digital camera. The set of picture frames can be supplied from the digital camera.
[016] In one embodiment, the generation of the refined set of camera orientation data includes an iterative optimization based on a gradient decrease.
[017] In one embodiment, the generation of the refined set of camera orientation data may include filtering based on a Gaussian filter.
[018] In one embodiment, the generation of the refined set of camera guidance data may include filtering based on a time derivative.
[019] In one embodiment, the camera orientation data set can include rotations, without any translations.
[020] In one embodiment, the camera orientation data set can include vectors, having both rotations and translations.
[021] In one embodiment, the generation of the refined set of camera guidance data is done by a social networking system.
[022] In one embodiment, the set of image frames, having associated time periods, and the camera orientation data set, having associated time periods, can be transferred to the network system only.
Petition 870160074035, of 12/08/2016, p. 9/66
5/54 by a user of the social networking system.
[023] In another embodiment of the invention, one or more non-transient computer-readable storage media incorporates software, which is operable when performed to conduct a process according to the invention, or any of the embodiments mentioned above.
[024] In another embodiment of the invention, a system comprises: one or more processors; and a memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing instructions to execute a process according to the invention, or any of the embodiments mentioned above.
[025] Many other features and embodiments of the invention will be evident from the attached drawings and the detailed description presented below.
BRIEF DESCRIPTION OF THE DRAWINGS [026] Figure 1 illustrates an exemplary model of a camera without a lens, according to one embodiment.
[027] Figure 2 shows an illustration of two exemplary camera orientations and their corresponding image planes, according to one embodiment.
[028] Figure 3 shows an illustration of an exemplified distorted image, captured by an RS camera, and the transformations to correct the image, according to one embodiment.
[029] Figure 4 illustrates an exemplary digital video stabilization module, according to one embodiment.
[030] Figure 5 illustrates graphs of examples of input data and resulting data, after refining, according to one embodiment.
[031] Figure 6 illustrates an example process for refining based on restrictions, according to one embodiment.
Petition 870160074035, of 12/08/2016, p. 10/66
6/54 [032] Figure 7 illustrates an exemplary network diagram of a system for modifying a set of digital video image frames, to produce a stabilized digital video within a social network system, according to one embodiment .
[033] Figure 8 illustrates an exemplary computer system, which can be used to implement one or more of the embodiments described in this specification, according to one embodiment.
[034] The figures illustrate various embodiments of the present invention for illustrative purposes only, in which the figures use similar reference numbers to identify similar elements. A person skilled in the art will easily recognize from the discussion presented below that alternative embodiments of the structures and processes, illustrated in the figures, can be employed without departing from the principles of the invention described in this specification.
DETAILED DESCRIPTION
DIGITAL VIDEO STABILIZATION AND ROTATING SHUTTER CORRECTION USING GYROS [035] Rolling shutter correction is a technique for removing image distortion produced by camera movement between frames. State-of-the-art cameras use charge-coupled device (CCD) sensors, which have a global shutter (GS). On a GS camera, including many DSLRs, all pixels in the CCD sensor are read and reconstructed simultaneously. Therefore, all pixels collect light during the same time interval. Consequently, the movement of the camera, during exposure, results in a bit of image obscurity on these devices. In comparison, cameras from previous generations typically make use of metal oxide semiconductor (CMOS) sensors. In particular, these sensors employ a camera orientation, in which the rows of images are read and reconstructed sequentially. This approach can re
Petition 870160074035, of 12/08/2016, p. 11/66
7/54 want fewer circuits, compared to CCD sensors, and can lower the cost of manufacturing CMOS sensors. For this reason, CMOS sensors are often used in cell phones, music players and some compact camcorders from previous generations. Sequential reading, however, means that each row is exposed during a slightly different time window. Therefore, the movement of the camera while reading the row will produce a distorted image. Fast moving objects will also look distorted.
[036] Image reading on an RS camera is typically in the range of milliseconds. Therefore, RS distortions are basically caused by high frequency camera movements. MIS systems can therefore be used to stabilize the camera. While this approach eliminates rolling shutter distortion, in practice, the price range and size of MIS systems make them unsuitable for RS cameras. Some digital rolling shutter rectification techniques can correct RS artifacts in a single image, but require user input. In comparison, some embodiments of the present invention can correct RS artifacts in single images, without requiring user input.
[037] For video, RS rectification in a frame sequence, using feature tracking to estimate camera movement from video, can present problems. Once the camera movement is known during an RS exposure, it can be used to rectify the frame. Since this approach is based on feature trackers, it has some disadvantages discussed previously in the case of video stabilization.
[038] Digital video stabilization techniques may include cropping or zooming in on video frames introduced. This can allow individual frames to be moved, rotated or distorted to neutralize unwanted deformations introduced by a communication establishment. The degree
Petition 870160074035, of 12/08/2016, p. 12/66
8/54 clipping can determine how much deviation (or wiggling space) is available to remove these deformations. If, for example, the frame is moved too far, empty regions (for example, regions that do not have pixel data) may be visible. Some embodiments of the present invention not only rectify the movement of the camera by neutralizing the communication establishment, but also do so while preventing the empty regions from being shown. It must be considered that other processes to provide the deviation for stabilization, unlike cropping or zooming, can also be implemented. For example, reconstruction techniques can be implemented to fill the empty regions introduced by stabilization. Reconstruction can be used in place of, or in addition to, cropping or zooming. For example, in one embodiment, a function can be implemented to determine whether a given deformation results in a frame with all the pixels renewed (whether by reconstruction, zooming, cropping, etc.) satisfactory or not.
[039] Systems and processes are provided to digitally stabilize videos by smooth camera rotation computation, which meet the requirements for preventing empty regions. This can enable maximum smooth camera rotations to be produced for a given cropping factor or zooming.
[040] The digital video stabilization process can start, for example, with the video being captured by a camera or a device including a camera, such as a mobile phone, a tablet, a watch, a clothing accessory, etc. The video can include several successive image frames, which are captured. The video may become unstable due to its size and weight. The rolling shutter used by the sensors in the camera can provide distortion in the transmitted image frames. Gyroscopes can be used to measure the movements (for example, rotations) of the camera when capturing video. The movement of
Petition 870160074035, of 12/08/2016, p. 13/66
9/54 measured camera can be used to stabilize the video and rectify the rolling shutter, to result in the stabilized video having output frames with corrected images.
[041] The techniques of the present invention can improve the video quality of RS cameras. In one embodiment, gyroscopes of microelectromechanical systems (MEMS) are implemented to measure camera speed. Other gyroscopes and motion detection devices can also be implemented. Gyro measurements can be used to perform video stabilization (for example, motion compensation between frames) and rolling shutter correction (for example, motion compensation between frames). This approach can be both computationally inexpensive and robust, which can make it particularly suitable for real-time deployments on, for example, mobile platforms.
[042] Systems and processes in a unified model of a rotating camera and a rolling shutter can use the model to compute a distortion, which simultaneously performs rolling shutter correction and video stabilization. The optimization techniques, which can be provided, automatically calibrate the gyroscope and the camera. This can allow recovery of unknown parameters, such as gyro shift and delay, as well as the focal length of the camera and the rolling shutter speed, of a single video capture and gyroscope. Therefore, any combination of gyroscope and camera hardware can be calibrated without the need for adjustment in a specialized laboratory. A device including the camera can also include a motion detector device, such as a gyroscope. For example, many smartphones have cameras and motion-detecting devices, such as gyroscopes and accelerometers. In this way, video stabilization and real-time shutter correction can be provided without requiring the use of
Petition 870160074035, of 12/08/2016, p. 14/66
10/54 feature trackers or MIS systems. What's more, inexpensive MEMS gyroscopes can be implemented to directly measure camera movement. Inertial measurement units (IMUs) can be used to defog the image and to assist a KLT feature tracker.
[043] The measurement of camera movement using motion detecting devices, such as gyroscopes, allows digital video stabilization and RS rectification to be performed with high computational efficiency. This approach can be robust, even in poor lighting or substantial background movement, because video content is not used for motion estimation. Furthermore, as mentioned above, many cell phones equipped with cameras are already equipped with gyroscopes or other motion-detecting devices. Compared to MIS systems, MEMS gyroscopes can be significantly less inexpensive, more versatile and less bulky.
[044] In one embodiment, video stabilization can be conducted in three stages: estimating camera movement; movement refining; and image distortion. Rolling shutter rectification can be conducted in a similar manner, except that effective camera movement can be used for distortion computation, rather than refined movement. As will be discussed in more detail in this specification, both video rectification and rolling shutter correction can be conducted on distortion computation under a unified structure.
[045] In one embodiment, the camera movement can be modeled in terms of rotations only. It must be considered that translations can be measured in addition to, or in place of, rotations in other embodiments. In some cases, translations can be difficult to measure precisely using, for example, IMUs. Furthermore, the accelerometer data may need to be integrated twice to obtain translations. In comparison, gyroscopes measure the speed of rotation. Per
Petition 870160074035, of 12/08/2016, p. 15/66
11/54 so much, gyroscopic data may only require a single integration to obtain the orientation of the camera. Consequently, translation measures may be significantly less accurate than guidance measures in some cases. Furthermore, translation measurements can be complicated by objects at different depths moving to different degrees. In one embodiment, the feature-based or stereo structure of motion algorithms (SfM) can be implemented to obtain depth information. Distortion frames, to remove translations, can be performed in some embodiments, but can be complicated by parallax or occlusions.
[046] The modeling of camera-to-system translations can have some consequences. For example, an optimizer may fall into a minimal location, while trying to reconstruct the translations of a feature tracker. An algorithm can assume that the camera is imagining a purely flat scene (for example, of constant depth). Therefore, the reconstruction of translation can be complicated, due to the parallax not modeled in the video.
[047] The embodiments modeling the camera rotation only in terms of rotations, or, basically, in terms of rotations, can minimize the problems encountered with translations. Camera shake and rolling shutter distortion occur mainly from rotations, since translations are attenuated quickly with increasing depth, and objects are far enough away from the lens, so that the translational instability of the camera does not produce a movement significant in the image.
EXAMPLIFICATIVE CAMERA MODEL [048] In one embodiment, a rotating shutter camera model, which is based on a model without a lens, is provided. Figure 1 illustrates a model camera without an exemplificative lens 101, according to one embodiment. A radius from a camera center c to an x point in the scene will intercept
Petition 870160074035, of 12/08/2016, p. 16/66
12/54 the image plane at point x. Therefore, the projection of latitude and longitude in the image plane depends on the center of the camera c, the focal length f and the location of the camera axis (Ox, _y ) in the image plane. In a camera without a lens, the relationship between the image point in homogeneous coordinates and the corresponding X point in latitude and longitude coordinates can be specified by the example equation (l).
x -KX.and X - AK '^ x (l) [049] In this case, λ is an unknown scale factor and K is the intrinsic camera. K ' ¹ can be specified by the example equation (2).
/ 1 0 & Λ
K ¹ - 0 1 ..... 4¼ (2) Ü 0 / / [050] (Ox, Oy) is the origin of the camera axis in the image plane and f is the focal length. The focal length of the camera is an unknown parameter, which can be retrieved. The camera can be considered to have square pixels and the upper diagonal entries set to 1. However, other embodiments can enlarge this model to include non-square pixels or other optical distortions.
[051] Distortion can occur from high frequency camera rotations. For example, high-frequency camera rotations, while the shutter is rolling from the top to the bottom, can cause the output image to appear distorted. This distorted image can be modeled mathematically.
[052] The origin of latitude and longitude can be adjusted at the origin of the camera. The movement of the camera can then be described in terms of its orientation R (t) at time t. Thus, for any scene point X, the corresponding image point x at time t can be shown by the example equation (3).
KH 'X. (3) [053] The rotation matrices R (t) and SO (3) can be computed by using
Petition 870160074035, of 12/08/2016, p. 17/66
13/54 combination of variations in camera angle Δθ (ΐ). The spherical linear interpellation (SLERP) of quaternions can be used to uniformly interpolate the orientation of the camera and prevent locking of cardanic suspension. The variation in angle between samples of the gyroscope can be small enough that the Euler angles also function as quaternions of rotation. Δθ (ΐ) can be directly obtained from rotation speeds ro (t) measured by gyroscope, as specified in the following example equation (4).
[054] tod is the displacement of the gyro and td is the delay between the gyro and the time periods of frame samples. These parameters are other unknowns, which can also be recovered by the model.
[055] The roller shutter can also be introduced in the camera model. On an RS camera, each row of image is exposed to a slightly different time. Camera rotations during this exposure can, however, determine image distortion. The translational instability of the camera, during exposure of the camera orientation, does not significantly affect image distortion, because objects are typically away from the lens. For example, if the camera tilts from side to side while the shutter is rolling, then the output image will be distorted, as shown in Figure 3. The time at which point x was transformed into an image, in the frame i, depends on how far down the picture is. It can be determined that x was transformed into an image at time t (i, y), as specified by the example equation (5).
/ (£ 3 /) - A 4- / _s * -y / ft, where x - (x, te (5) [056] y is the image row corresponding to point x. H is 0 total number of rows in the frame, ti is the time period of the i ^Q frame. The term ts indicates that the further away in the frame, the longer it takes for the rolling shutter to appear in that row. Therefore, ts is the time it takes to read a full frame
Petition 870160074035, of 12/08/2016, p. 18/66
14/54 going from row to row, from top to bottom. A negative ts value will indicate a rolling shutter that moves from the bottom to the top. The automatic recovery of the signal and the value of ts is described in more detail in this specification.
[057] The relationship between the image points in a pair of frames for two different camera orientations can be obtained. Figure 2 illustrates a graphical representation of two camera orientations and their corresponding image planes, according to one embodiment. Graphical representation 201 includes two camera orientations 202 and 203. Camera orientation 202 includes image plane i. Camera orientation 203 includes the image plane j. An image of scene point X appears in the two frames in which radius 211 intersects the image planes i and j. For a scene point X, the projected points xí and xj, on the image plane of the two frames i and j, respectively, can be specified by the following example equations (6).
x _; - KR (<A _fc )) X, and Xj = KRÒÜ, y ₃ nx (6) [058] If these equations are redrawn and if X is replaced, a mapping of all points in the table was all points in table j is obtained, as specified by the following example equation (7).
xj kli ·. ·, K '·'> K 'x (7) [059] Although the relationship between the two frames has been described in relation to the same video, in other embodiments, the frames can be mapped from a camera, which rotates according to R (t), in another camera, which rotates according to R '(t). In one embodiment, both camera centers can be considered to be at the origin. The distortion matrix W, which maps the points from one camera to the other, can be specified according to the following example equation (8).
Web, te) - Kl <(8)
Petition 870160074035, of 12/08/2016, p. 19/66
15/54 [060] Equation 7 can then be specified more compactly according to the following example equation (9).
where IT ~ R (9) [061] W depends on both the yi and yj image rows of the xi and xj image points, respectively. This distortion matrix can be used to compare the points in frame i to the corresponding points in frame j, while considering the effects of the rolling shutter in both frames.
[062] This formulation of a distortion matrix provides a rolling shutter correction and video stabilization. A synthetic camera can be created, which has uniform motion and a global shutter. This camera movement can be computed by applying a Gaussian low-pass filter, for example, for the movement of the input camera, which results in a new set of rotations R '. The duration of the rolling shutter ts for the synthetic camera can be set to 0 for an overall shutter. W (ti, t (i, yi)) can then be computed in each row of image yi in frame i at the moment, and the distortion can be applied in that row. The first term of W can then only depend on the time of frame ti. This operation can map all input frames on the synthetic camera, and therefore simultaneously remove rolling shutter distortion and video shaking.
[063] In certain embodiments, W (ti, t (i, y,)) is not computed for each row of image yi. Instead, the input image can be subdivided and the distortion computed in each vertical subdivision. Figure 3 illustrates an exemplary transformation to correct the distortion, according to one embodiment. The distorted input image frame 301 shows a subdivided distorted image, which was captured by an RS camera. The distortion is computed in each vertical subdivision, as shown in picture frame 311.0 Picture frame 311 shows a linear approximation in the form of parts of non-linear distortion. As shown
Petition 870160074035, of 12/08/2016, p. 20/66
16/54 by the resulting image frame 316, several numbers of subdivisions may be sufficient to eliminate the artifacts. For example, in one embodiment, 10 subdivisions may be sufficient to eliminate visual artifacts. A distorted mesh of the input image was created, which is a linear approximation in the form of parts of the non-linear distortion. While ten subdivisions may be sufficient to remove any visible RS artifacts, other embodiments may include a different number of subdivisions. The sampling approach can be referred to with a reverse interpellation. Reverse interpolation can be easy to implement in a graph processing unit (GUI) using vertex shading units. The GPU fragment shading unit can take on resampling the mesh distorted image using bilinear interpolation. The RS distortion in real videos may not be significant enough to produce the artifact outline, due to the bilinear inverse interpolation. Therefore, reverse interpolation can work well in practice. The shutter correction using global image distortions may consider the camera rotation to be more or less constant during the exposure of the shutter. A linear approach may fail to rectify the rolling shutter, as shown by picture frame 306 in Figure 3.
CAMERA AND GYROSCOPE CALIBRATION [064] Calibration techniques are provided to recover the unknown parameters of the camera and gyroscope described in this specification. Calibration can provide direct W computation of the gyroscope data. Unknown parameters in the model, described in this specification, may include: the focal length of the camera f; the duration of the rolling shutter t _s ; the delay between the gyroscope and the time frames of tu frame samples; and the wd gyro shift.
[065] In some cases, one or more of these parameters, such as length
Petition 870160074035, of 12/08/2016, p. 21/66
17/54 focal length f of the camera, can be specified by the manufacturer. In some cases, these parameters can be measured experimentally. For example, a quick flashing display can be used to measure the duration of the rolling shutter ts. However, these techniques can tend to be inaccurate and error prone. These techniques can also be tedious. The duration of the rolling shutter can typically be in the range of milliseconds. Therefore, a small misalignment in td or t _s can cause failure in the rectification of the rolling shutter.
[066] In one embodiment, these parameters can be estimated from a single video capture and gyroscope. For example, the user may be recording a video and a gyro trace while standing still and shaking the camera while pointing at a stationary object, such as a building. The length of the video clip can vary in different embodiments. In one embodiment, a short video clip (for example, ten seconds or less in length) may be sufficient to estimate all unknown parameters. This calibration step may need to be done only once for each camera and gyroscope layout.
[067] In one embodiment, the association points are identified in consecutive video frames. Association points can be identified using, for example, the invariant scale characteristic transform (SIFT). Wandering parts can be discarded using, for example, a global sampling consensus (RANSAC). The result can be a set of matching points xí and xj for all neighboring frames in the captured video. In view of this substantiated truth, the calibration can be formulated as an optimization problem, in which the error of reprojection of the least squares of all correspondences of points can be minimized. This is specified in the following example equation (10).
v x ‘‘. X (W)
Petition 870160074035, of 12/08/2016, p. 22/66
18/54 [068] Several non-linear optimizers can be used to minimize the objective function. The decrease in coordinates by the direct objective function can converge quickly, and is implemented in one embodiment. Each time a step is considered when objective function J does not decrease, the step direction is reversed and the step size of the corresponding parameter is decreased. The algorithm can end when the step size, for all parameters, falls below a desired threshold, just as a desired precision is obtained. Convergence can occur quickly in some cases. For example, in one embodiment, convergence can occur in 2 seconds or less for a calibration video lasting about 10 seconds.
[069] In one embodiment, optimization can be initiated by adjusting the focal length so that the camera has a 45 ^Q field of view. All other parameters can be set to 0. With these initial conditions, the optimizer can converge on the correct solution for the data set. More generally, falling into a minimal location (for example, when the delay between the gyroscope and frame time periods is long) can be avoided by resetting the coordinate-lowering algorithm for a plausible range of parameters, and section of the best solution. The average reprojection error for the parameters retrieved correctly can be, for example, around 1 pixel.
[070] An additional unknown parameter in the model may be the relative orientation of the gyroscope in relation to the camera. For example, the rotations about the y-axis of the gyroscope can correspond to the rotations about the x-axis of the camera.
[071] To find out the orientation of the gyroscope, the 3 axes of rotation can be exchanged, and the optimizer can be executed for each exchange. The exchange that best minimizes the lens may correspond to the order of the camera's axes. The reprojection error can be significantly higher for in-house exchanges
Petition 870160074035, of 12/08/2016, p. 23/66
19/54 correct. Therefore, this process can work well in practice.
[072] Although the camera was considered to have a vertical rolling shutter, the RS model can be easily modified to work with image columns, instead of rows. The discovery of the minimum reprojection error, for both cases, can indicate whether the camera has a horizontal or vertical rolling shutter.
[073] Finally, the results obtained through calibration can be demonstrated by analyzing the video signals and the gyroscope, before and after calibration. Considering the small rotations between the frames, the translations in the image can be approximately computed for rotation, as specified in the following example equation (11).
x (/ 'i «f * á (t -Í-ÍA where J ^x = LM /) _{(] n} ''[ú = 1¾.¾)' [074] Equation (11) considers that there are no effects due to the rolling shutter (for example, ts = 0), and the rotations around the z axis (for example, ω _ζ ) can be ignored, x is the average translation speed along x and y for all point matches in consecutive frames. If the optimizer has converged to the focal length f and the gyroscope delay to, then the two signals must be aligned. Before calibration, the amplitudes of the x and f * ωγ (t + íd) signals are not the same, because the initial estimate for f is too low The signals can be shifted when td is started at 0. After calibration, the signals can be well aligned, because the precise focal length and gyro delay have been recovered. Precise gyroscopes, such as gyroscopes MEMS, allow gyroscopic data to be associated with image movements, resulting in improved video stabilization and rolling shutter correction.
ROTATION REFINING BASED ON RESTRICTION [075] In some aspects of the present invention, the system and processes
Petition 870160074035, of 12/08/2016, p. 24/66
20/54 can be provided to compute an optimally uniform camera movement, according to the constraint, that the empty regions are not visible, or below a minimum threshold value. Figure 4 illustrates a digital video stabilization module, according to one embodiment. The digital video stabilization module 400 is shown including an input module 401, a refinement module
402 and a distortion module 403.
[076] Input module 401 can provide inputs to stabilize refinement module 402 and distortion module 405. Input module 401 can receive inputs associated with the video to be stabilized. For example, the entries may include a set of N Fi frames, corresponding to ti times for the N Fi frames, and camera 0i directions, where i = {1 ..N}.
[077] The refinement module 402 comprises a set of new refined camera orientations φ, so that a constraint function f (φ, t) is satisfied. The refinement module 402 may include a rotation speed module
403 and a 404 constraint determination module.
[078] The rotation speed module 403 computes the rotation speed, to ensure that a rotation speed sufficiently small is maintained for the generation of refined camera orientations. The constraint determination module 404 determines whether a constraint is satisfied for the desired orientations φί at time ti. For example, in one embodiment, the constraint function f (φ, t) can return 0 or 1, depending on whether the empty regions are visible or not, respectively, in view of a desired orientation φί at time ti.
[079] For example, the 404 constraint determination module can determine whether a minimum number of empty regions (for example, below a threshold amount) are produced in successive image frames. If the number of empty regions falls below the threshold amount, then the constraint is satisfied
Petition 870160074035, of 12/08/2016, p. 25/66
21/54 made (for example, the number of empty regions does not exceed the threshold amount), and the desired orientation ψ and its corresponding time t can be used to generate a refined orientation, which has a sufficiently small rotation speed, which does not generate a number of empty spaces above a threshold value. If the constraint is not satisfied (for example, the number of empty regions exceeds the threshold amount), then the desired orientation φ can be adjusted to maintain a sufficiently small rotation speed while satisfying the constraint. In one embodiment, the threshold amount for empty regions is zero. In another embodiment, the threshold amount of empty regions is approximately zero or a negligible value, which is determined to be undetectable by the human eye. In yet another embodiment, the threshold amount of empty regions is a value that prevents cropping, zooming or reconstruction from being used effectively, such as to eliminate all empty regions, or approximately all empty regions. In other embodiments, the threshold amount of empty regions can be adjusted as desired, based on the application and the tolerance level.
[080] Distortion module 405 generates distorted frames, based on the set of new refined orientaçõesί camera orientations computed by refinement module 402. For example, distortion module 405 can implement a distortion function g (F, φ, t), which adopts as an input a frame F, a refined orientation φ and its corresponding time t, and generates a distorted frame F '. In view of the uniform variation of φί over time, as transmitted by the refinement module 402, and of appropriate selections of the f and g functions, the resulting distorted frames F'i will compose a stabilized output video. The function f can depend on the selection of the distortion function g, and can simply indicate whether, after applying the distortion function g, the empty regions will be visible or not in the frame.
[081] The specific distortion function implemented may vary in different
Petition 870160074035, of 12/08/2016, p. 26/66
22/54 close achievements. The different distortion functions can be suitable for the various cameras or the desired approaches. For example, the distortion function implemented can be based on whether a camera has a rolling shutter and small lens aberrations, or whether it has a global shutter. For example, in one embodiment, for cameras with a global shutter, a homography distortion function can be implemented. Other approaches, such as related transformations, or a rotation plus a translation in the frame image space, can be implemented.
[082] In one embodiment, the Oi entry are rotations in the SO (3) group. There are several ways to represent rotations, such as by means of matrices and rotation quaternions. Representations that are based on SO (3) can be converted into a representation that facilitates uniform interpellation, such as quarters instead of Euler angles. Oi rotations can be computed from an image-based feature tracker, for example, or directly by measuring and integrating gyro readings. Any other process, which produces accurate estimates of the camera orientation, can be implemented in other embodiments.
[083] Although Θ has been described in terms of camera rotations, in other embodiments, Θ can include a vector, which contains both camera rotations and translations in 3D space. For example, vectors, which include both rotations and translations, can be produced by a motion algorithm structure. In one embodiment, Θ may include translations or rotations in the frame image space, or other less accurate but potentially computationally cheaper approximations. As long as Θ can be uniformly challenged and the resulting ο Θ can be introduced into a corresponding f and g function, digital video stabilization can be achieved.
[084] The term uniform camera movement can be used, at present
Petition 870160074035, of 12/08/2016, p. 27/66
23/54 descriptive report, to refer to small variations in the speed of rotation. This is different from small variations in the rotation of neighboring frames. Small variations in the speed of rotation can produce guidelines that facilitate the entry and exit of restrictions imposed over time. Small variations in rotation of neighboring frames promote interpellation for and from constraints, while producing discontinuities in time-derived derivatives when the constraint is met.
[085] Restriction-based rotation refining can include an optimization, which includes minimizing an energy function, based on the speed of rotation and the constraint.
[086] In one embodiment, the energy function, J, to be minimized, can be specified by the following example equation (12).
[087] Rotations rot can be represented as unitary quaternions (also known as versors). Furthermore, the rigid constraint can be replaced by a smooth constant, as specified in the following example equation (13).
You can determine how intensely constraint f is met. For example, specifying λ as infinite can ensure that no empty regions are visible, provided that the restrictions have been satisfied.
[088] Several optimization algorithms can be used to minimize the energy J function in any form shown in the example equation (12) or (13). In one embodiment, an iterative algorithm based on gradient decrease of the example equation (12) is implemented, when the constraint can be met in each iteration. In one embodiment, frames can be spaced
Petition 870160074035, of 12/08/2016, p. 28/66
24/54 temporally likewise, such as a camera, which records several specific frames per second. With the tables equally spaced, the denominator can be considered to be constant and can be eliminated from the sum. The derivative can then be specified by the following example equation (14).
[089] C can be a constant, which controls the magnitude of the gradient. In some embodiments, this can be established automatically by some forms of gradient decrease, such as moment-based processes. In other embodiments, this can be set as desired to control the rate of decrease in a common gradient decrease. An exemplary value can be, for example, C = 2 / (Δΐ) ² . Equation (14) can be specified more compact by the following example equation (15).
[090] The K nucleus can be a Laussian transform of a Gaussian filter (LoG). The LoG filter can be approximated with a Gaussian Difference Filter (DoG) or Box Difference (DoB). The k-core can also be adjusted by interlacing a LoG filter with a Gaussian filter. It can gradually control the speed of rotation in the event of variation (or the amount of restrictions reached or not). The selections of the LoG / DoG / DoB and / or Gaussian filters can affect the coefficients and size of the K nucleus, but since the nucleus comprises a form of a time derivative, the optimized orientations may present some form of achieving or not the restrictions .
[091] In one embodiment, applying the kernel to the quaternions is done as a linear weighting of the 4D vectors. In theory, interpellation of quaternions using spherical linear interpolation (slerp) can be an accurate process. For
Petition 870160074035, of 12/08/2016, p. 29/66
25/54 small variations in angles, linear interpolation plus a normalization of the resulting quaternion (lerp), in each iteration, is sufficient to produce sufficiently accurate results. These results can be obtained because sen (θ) ~ Θ, when Θ is small. The variation in the angle induced by the communication establishment (for example, variations that are going to be rectified) is not typically large. Furthermore, reasonable levels of cropping, zooming or reconstruction factors may not leave much margin. Only small angles may not produce empty regions. Thus, the angle induced by the establishment of communication between quarternions of consecutive orientations may not be, in practice, large, and the approximation can be both precise and computationally efficient. In addition, for fast camera rotations (by fast horizontal movement over the image on the left), the restriction function f can dominate the resulting orientations. In this way, any lerp inaccuracies may not be noticeable.
[092] In one embodiment, the restriction-based rotation refine can be specified in terms of the following exemplificative algorithm (1).
Algorithm 1 - Constraint-based rotation refinement for «= l.JV do ό ΐ A end for for j = L.mí-miíer do for / - 3 .. (A - 2) d« if f (n ( 4s - <7ΚΦι), = 0 then.
<- - ϋΚΦϊ) end if end for end for [093] φί can be pre-computed in the outer loop, so this does not change the inner loop. The number of iterations, numiter, can be set high enough to result in uniform rotations. The resulting pode φί can then be gas powered to produce a stable output video.
[094] η (φ) is the normalization step in lerp and can be defined as: η (φ)
Petition 870160074035, of 12/08/2016, p. 30/66
26/54 = φ / | φ |. The indexing i in algorithm 1 is selected so that φι is valid, and therefore can depend on the size of the k kernel. In this example, the revs at the limits can be kept fixed. In other embodiments, the limit can be extended through extrapolation, allowing the entire set of guidelines to be iterated during optimization. The entire set of guidelines can be specified as the following set.
[095] Figure 5 illustrates graphs of examples of input data and resulting data after refining, according to one embodiment. The top graph includes the input data 502 represented against time, with the restrictions 503 indicated by circles. The background graph includes the resulting data 504, including restrictions 503, which resulted from the input data 502, after refining, according to the restriction-based route refining techniques described in this specification. For example, the resulting data 504 can be refined by imposition so that the rotation speed (or derivative) is small (or below a threshold value), while ensuring that the constraint (for example, a threshold amount of empty regions) that are not produced in successive image frames) is satisfied. Small variations in the speed of rotation can produce guidelines that may or may not follow the restrictions imposed over time. The resulting data 504 does or does not follow restrictions 503. On the other hand, the intermediate graph includes the resulting data 506, including restrictions 503, which resulted from an attempted refining of input data 502 by imposing that the change in frame orientation neighbors is small. Small variations in rotation of neighboring frames interpolate in and out of constraints, while producing discontinuities in time-derived derivatives, on which the constraint is imposed. As shown, the resulting data 506 includes discontinuities in the derivative at restrictions 503.
Petition 870160074035, of 12/08/2016, p. 31/66
27/54 [096] Figure 6 illustrates an exemplary process for refining based on constraint, according to one embodiment. In block 601 of process 600, video data is received. The video data can include a set of image frames, having associated time periods. In one embodiment, block 601 can be executed by input module 701 of Figure 4.
[097] In block 603, the camera orientation data, having associated time periods, are received. For example, the device including the camera may also include an orientation sensor, such as a gyroscope, an accelerometer, etc., which generates camera orientation data, which tracks the camera's orientation, during video capture. The camera orientation data can include associated time periods, to connect or otherwise associate the camera orientation data with the set of images in the video data. In some cases, camera orientation data may be received at the same time as video data, such as in conjunction with video data. In one embodiment, block 602 can be executed by input module 701 of Figure 4.
[098] In one embodiment, blocks 601 and 603 can be executed by the device having the camera (for example, smartphone or other portable device), which is used to capture the video. For example, video data and camera orientation data can be received by capturing the video. In another embodiment, blocks 601 and 603 can be executed by a separate device (for example, a computer), which subsequently receives the video data captured by the device including the camera (for example, smartphone). For example, video data and camera orientation data can be transmitted or transferred to a device separate from the device including the camera and the orientation sensor, such as a smartphone with camera.
[099] In block 605, a set of camera orientation data is generated by minimizing the speed of rotation between successive image frames
Petition 870160074035, of 12/08/2016, p. 32/66
28/54 gem, minimizing (or limiting) the amount of empty regions in a refined set of resulting image frames. The resulting refined set of image frames is reoriented based on the refined data from the camera orientation data.
[0100] In one embodiment, the camera orientation data set is generated by minimizing any of the equations (12) or (13) described in this specification. In one embodiment, an iterative algorithm based on decreasing the gradient of the example equation (12) is implemented, in which the constraint can be imposed on each iteration.
[0101] In block 607, the image frame set is distorted to form a corrected image frame set. The image frame set can be distorted based on the associated time periods for the image frame set and the refined camera orientation data set. In one embodiment, an individual picture frame can be divided into several subsections, with each subsection having an associated time frame and camera orientation. Each subsection can be realigned based on the associated time frame and camera orientation to form an individual corrected image frame.
REDUCING THE VISIBILITY OF MOBILE BLURRED ARTIFACTS [0102] Digital video stabilization of video, captured in low light intensity, often produces blurred artifacts of strange motion. This may be because the moving blur looks strange when the movement (for example, a communication establishment) that caused the fact is removed. In some cases, it may be necessary to leave just enough of the communication establishment in the stabilized video, to explain the traces of the movement. If there is a trace of clear horizontal movement in a frame, it may be necessary for the orientation to vary horizontally according to that trace, for the trace to make
Petition 870160074035, of 12/08/2016, p. 33/66
29/54 sense. If there is no horizontal movement, then this trace can be added or removed without any reason in the stabilized video, which can cause blurred artifacts of visible movement.
[0103] In one embodiment, the change in orientation Δθί, which occurred while the camera shutter was open, is computed according to the following example equation (16).
- * - Y s. S . (16) [0104] ti ^s can represent the time in which the shutter is open for the Fi frame. ei is the duration of exposure of the frame. 0 (t) is the orientation of the camera at time t, which can be computed by interpellation using the following expression.
{bAlíL [0105] In the example described above for digital video stabilization and rolling shutter correction using gyroscopes, Δθί can also be computed directly from the gyroscope readings only by integration for the period in which the shutter is open. Equation (1) or equation (2) of internal sum can be modified as specified in the following example equation (17).
Al _Si . . . ·; | 2
V6 DO-L · ~ 04 íi> «. - í7 »i - 1. ....;
H ---------: --- 1 - (ΔΑ, - ΔΑ.-]) H i = 2 ”'- - (17) [0106] Equation (17) considers that ti ^s > ti and that (ti ^s + ei) <ti + i. The shutter does not open before the frame timeout, and closes before the next frame starts. In another embodiment, entry time periods can be calculated differently (for example, ti is the time when the shutter was closed), and a pre-processing step can be added to adjust time periods, so that requirements be satisfied.
[0107] In the embodiment described above, changes in orientation, which leave traces of movement in the frames, are maintained. In another embodiment, an approximation can be implemented by the following example equation
Petition 870160074035, of 12/08/2016, p. 34/66
30/54 (18).
[0108] The approximation in equation (18) can try to optimize φί, so that the changes in orientation are located along the traces of motion Δθί. The scale factor γ can control the compromise between refining and how closely the traces of motion must be followed.
SELECTING A GOOD ZOOM FACTOR [0109] If, for example, a tolerable zoom factor (or crop factor, reconstruction factor or any other measure applicable to the algorithm used to provide margin for digital video stabilization) is between 1 and 1.25x. For a given video, it may be desirable to determine the smallest zoom factor, which provides sufficient margin to produce smooth camera movements. In one embodiment, the uniformity of the resulting camera movement can be measured by the following example equation (19).
r- τ-Χϊ y / ü / M ΟΚΦόΛ) At -Λ <, ..
-3 (19) [0110] In equation (19), q measures how often the empty region constraint is imposed. For example, a value of q = 0.1 may mean that the average f is imposed every 10 frames, to prevent empty regions from being shown. If, for example, a spacing of 20 frames is desired to follow the restrictions or not, then a good value can be q = 0.05.
[0111] To find the zoom factor z ', which provides a desired q' value, algorithm 1 can be solved for a range of zoom factors (ie, z ε [1, 1.05, 1.1 , 1.15, 1.2, 1.25]). Assuming a resulting zoom factor z and ο φί obtained by executing algorithm 1, the resulting q value can be computed from equation (19). The zoom factor, which provided the q closest to q ', can then be selected. In another embodiment, the zoom factor z 'can be determined
Petition 870160074035, of 12/08/2016, p. 35/66
31/54 by linear interpolation of the resulting data set (q, z) in q '. The check table (q, z) for typical portable records can also be pre-computed and q 'estimated from the median or average rotation speed. This approach may be less accurate for a particular video, but it may be faster because it does not require you to run the optimization algorithm multiple times.
IMPROVING COMPUTATIONAL EFFICIENCY [0112] There are several ways to improve the efficiency of the algorithm (1), such as introducing approximations, using different gradient decrease optimizers, etc. In one embodiment, the size of the K-core in equation (15) can be reduced and the gradient can be updated as specified in the following example equation (20).
â.7 —— = Ο '(2ΚΦι-ΚΦί-ι -ΚΦμι j ^where K (20) [0113] By reducing the kernel size (and reusing it), the gradient computation can be effectively accelerated. Another kernel property in equation (20) is that the value of Κφί can be computed more precisely using slerp, which can be done by using evenly spaced frames, as specified in the following example equation (21).
_q :, A - õ-i.
~ Ç>, _t - Sl & Tpl - -------------, i
A + i ..... <'i η n [0114] Even more precisely, the spherical tangent formed by φί and slerp can be determined and used in equation (20). The spherical tangent formed by φί and slerp can be specified by the following example expression.
...__ £ · ... .s
Φ * and 15 Φ - /. 4-1 · _t . * _{1 t} ) [0115] In another embodiment, efficiency can be improved by improving the convergence rate of algorithm 1, described in this specification. This can be achieved by running it from coarse to fine.
Petition 870160074035, of 12/08/2016, p. 36/66
32/54
For example, the algorithm may be initialized by 1 , and supplying each solution to 16 ^to φί orientation. Linear or cubic interpellation (for example, slerp or squad) can then be used to compute φί between them. A value of 8 for each orientation is obtained. The algorithm 1 can run again, but then optimizing every 8 ^â guidance for refining and restrictions. This can be repeated until algorithm 1 has been executed for each of the orientations.
[0116] The execution of coarse to fine can allow algorithm 1 to be executed for significantly less iterations in each step. The overall movement of the camera can be smoothed, and then the orientations can be refined to satisfy constraints at increasingly smaller frame intervals, while still maintaining a uniform still position. In another embodiment, non-uniform sampling can be used. For example, instead of harvesting every 16 ^Q frame, frames can be captured based on how far the orientation has been deviated from the previously captured frame. The segments can then be subdivided until a refined orientation has been computed for all frames.
[0117] Since coarser or non-uniform sampling can introduce large consecutive variations in orientations, the use of the slerp modification, presented in equation (21), can produce more accurate results.
EXAMPLE APPLICATIONS FOR REAL-TIME STABILIZATION [0118] Below is an exemplary embodiment of adopting the algorithm for real-time adjustments. With N frames kept in memory, the algorithm is executed in a sliding window mode. The index i can denote the beginning of the sliding window, and i = N can denote the end. The orientations inside the sliding window can be malleable, because they can be updated by executing the algorithm. These guidelines can be specified by the set (2)
Petition 870160074035, of 12/08/2016, p. 37/66
33/54 presented below.
[0119] The orientations preceding the sliding window (for example, φη) can be fixed. The starting orientation can be φί - 1 and the rotation speed (for example, measured by the gyroscope or computed from a feature tracker) can be integrated, to obtain the orientations {θί, θί = ι, ..., θι + η}. This can be used as input for the optimization algorithm and for computing {φί, φί + ι, ..., 1 í + n-2}. The orientations {φί-2, φί-i} and {φί + Ν-ι = φί + Ν-ι, φί + Ν = Φϊ + ν] can be kept fixed and can serve as the boundary conditions, which guarantees smooth movements for in and out of the sliding window.
[0120] Once the algorithm is executed, the orientation of φί, which can be used in the distorted frame Fi, can be obtained. Once the frame has been distorted, it can be passed together to the encoder and removed from the N frame buffer. A new frame can then be received from the camera, and the sliding window can be advanced by a frame.
[0121] The procedure can be repeated until recording is stopped and all frames have been processed. For the first of the tables, i = 1, {φ-ι, φο} can be undefined, φ-ι and φο can be adjusted to identify the quartenium. Once the last frame is received, the temporary storage can be emptied using the guidelines inside the final sliding window, to distort their respective frames.
[0122] In some cases, a small temporary storage (for example, N = 5) may not allow restrictions to be followed or not. In such cases, it may be necessary to modify the constraint function f to be a smooth constraint, which is raised to a hard constraint. This can be referred to as a function Γ (φ, t). Instead of returning 0 or 1, f returns a new 'y' orientation, which can be pushed from the edge of the frame. The closest orientation φ is ori
Petition 870160074035, of 12/08/2016, p. 38/66
34/54 to show an empty region, the more f can push the resulting φ 'away from the empty region. Algorithm 1 can then be reformulated as specified in the following example algorithm (2).
Algorithm 2 - Rotation refinement based on smooth constraint for I - l.JV of
Φ; φ— Oi end for for J = l..nu: miter do for i - 3 .. (IV - 2) do
Ά 4— / ίιηξφί - ti) end for end is [0123] Note that if f simply returns φί, when the orientation η (φί ΟΚφί) results in empty regions, then algorithms 1 and 2 can be equivalent. The optimization rate of algorithm 1 can be improved by using algorithm 2, with a function f that slerpa φί in the direction of η (φί - ΟΚφφ, but stops just at the point at which the empty regions are about to appear.
SOCIAL NETWORK SYSTEM - EXAMPLE IMPLEMENTATION [0124] Figure 7 is a network diagram of an exemplary 700 system for estimating user attention on a website or application, in accordance with an embodiment of the invention. System 700 includes one or more user devices 710, one or more external systems 720, a social network system 730, and a network 750. In one embodiment, the social network system, discussed in conjunction with the embodiments presented above, it can be implemented as the social networking system 730. For illustrative purposes, the embodiment of system 700, shown in Figure 7, includes a single external system 720 and a single user device 710. However, in other embodiments, system 700 may include more surface devices 710 and / or more external systems 720. In certain embodiments, the social network system 730 is operated by a social network provider, while external systems 720 are separated from the social network system 730 by fact that they can be operated by different entities. In several
Petition 870160074035, of 12/08/2016, p. 39/66
However, the social networking system 730 and external systems 720 operate together to provide social networking services to users (or members) of the social networking system 730. In this sense, the social networking system 730 provides a platform or structure, which other systems, such as external 720 systems, can use to provide social networking services and user functionality over the Internet.
[0125] User device 710 comprises one or more computing devices, which can receive input from a user and transmit and receive data over the 750 network. In one embodiment, user device 710 is a conventional computer system, which performs, for example, an operating system (OS) compatible with Microsoft Windows, Apple OS X and / or a Linux distribution. In another embodiment, the user device 710 can be a device having a computerized functionality, such as a smartphone, a tablet, a personal digital assistant (PDA), a mobile phone, etc. User device 710 is configured for communication over network 750. User device 710 can run an application, for example, a browser application, which allows a user of user device 710 to interact with the social networking system 730. In another embodiment, the user device 710 interacts with the social networking system 730 through an application programming interface (API), provided by the native operating system of the user device 710, such as iOS and ANDROID. User device 710 is configured to communicate with external system 720 and social system 730 over network 750, which can comprise any combination of local area and / or wide area networks, using communication systems with and / or wireless.
[0126] In one embodiment, the 750 network uses standardized communications, technologies and protocols. Thus, the 750 network includes links using technologies, such as 802.11 Ethernet, worldwide interoperability for microPetition access 870160074035, from 12/08/2016, p. 40/66
36/54 waves (WiMAX), 3G, 4G, CDMA, GSM, LTE, digital subscriber line (DSL), etc. Similarly, network protocols used on the 750 network may include switching multiple protocol labeling (MPLS), transmission control protocol I Internet protocol (TCP / IP), User Datagram Protocol (UDP), hypertext transport (HTTP), simple mail transfer protocol (SMTP), file transfer protocol (FTP) and the like. The data exchanged by the 750 network can be represented using technologies and / or formats, which include hypertext markup language (HTML) and extensible markup language (XML). In addition, all or some links can be encrypted using conventional encryption technologies, such as secure socket layer (SSL), transport layer security (TLS) and Internet Protocol security (IPsec).
[0127] In one embodiment, user device 710 can display content from external system 720 and / or social network system 730 by processing a markup language document 714, received from external system 720 and the social network system 730, using a browser application 712. The markup language document 714 identifies content and one or more instructions describing the formatting or presentation of the content. By executing the instructions included in the markup language document 714, the browser application 712 displays the identified content using the format or presentation described by the markup language document 714. For example, the markup language document 714 includes instructions for generate and display a network page having multiple frames, which include text and / or image data retrieved from external system 720 and social network system 730. In various embodiments, markup language document 714 comprises a data file, including extensible markup language (XML) data, extensible hypertext markup language (XHTML) data, or other data
Petition 870160074035, of 12/08/2016, p. 41/66
37/54 markup language. In addition, the markup language document 714 may include JavaScript Object Notation (JSON) data with padding (JSONP), and JavaScript data to facilitate data exchange between the external system 720 and the 710 user device. browser 712 on user device 720 can use a JavaScript compiler to decode the markup language document 714.
[0128] The markup language document 714 may also include, or be linked to, applications or application frameworks, such as FLASH ™ or Unity ™ applications, the SilverLight ™ application framework, etc.
[0129] In one embodiment, user device 710 also includes one or more cookies 716, including data indicating whether a user of user device 710 is registered on the social networking system 730, which may lead to modification of the reported system data from social networks 730 to user device 710.
[0130] External system 720 includes one or more network servers, which include one or more network pages 722a, 722b, which are communicated to user device 710 using network 750. External system 720 is separate from the network system social 730. For example, external system 720 is associated with a first domain, whereas social system 730 is associated with a separate social network domain. Network pages 722a, 722b, included in external system 720, comprise markup language documents 714, which identify content and which include instructions specifying formatting or presentation of the identified content.
[0131] The social networking system 730 includes one or more computing devices for a social network, including multiple users, and providing social network users with the ability to communicate and interact with other social network users. In some cases, the social network can be represented by a graph, that is
Petition 870160074035, of 12/08/2016, p. 42/66
38/54 is, a data structure including borders and nodes. Other data structures can also be used to represent the social network, including, but not limited to, databases, objects, classes, meta-elements, files or any other data structures. The 730 social networking system can be administered, managed or controlled by an operator. The operator of the 730 social networking system can be a person, an automated application, or a series of applications to manage content, regulate policies and collect usage metrics within the 730 social networking system. Any type of operator can be used.
[0132] Users can join the social networking system 730 and then incorporate connections to any number of other users of the social networking system 730, to which they want to be connected. As used in this specification, the term friend refers to any other user of the social networking system 730, with which a user has established a connection, association or relationship with the social networking system 730. For example, in one embodiment, if users in the 730 social network system are represented as nodes in the social graph, the term friend can refer to a border formed between, and directly connecting, two users.
[0133] Connections can be added explicitly by a user or can be created automatically by the social networking system 730, based on common characteristics of users (for example, users who are students of the same educational institution). For example, a first user specifically selects another private user to be a friend. Connections in the social networking system 730 are usually in both directions, but need not be, so the terms user and friend depend on the frame of reference. Connections between users of the 730 social networking system are usually bilateral (two-way), or mutual, but connections can also be unilateral, or one-way. For example, if Bob and Joe are both system users
Petition 870160074035, of 12/08/2016, p. 43/66
39/54 social media theme 730 and connected to each other, Bob and Joe are connected to each other. If, on the other hand, Bob wants to connect to Joe to view data communicated to Joe's 730 social networking system, but Joe doesn't want to form a mutual connection, a one-way connection can be established. The connection between users can be a direct connection; although, some embodiments of the social networking system 730 allow the connection to be indirect through one or more levels of connections or degrees of separation.
[0134] In addition to establishing and maintaining connections between users and allowing interactions between users, the 730 social networking system endows users with the ability to promote actions on various types of items supported by the 730 social networking system. These items can include groups or networks (that is, social networks of people, entities and concepts) that users of the 730 social networking system may belong to, event entries or calendar that a user may be interested in, computer-based applications that a user can use through the 730 social networking system, transactions that allow users to buy or sell items through services provided by or through the 730 social networking system, and interactions with advertising, which a user may or may not perform on the social networking system 730. There are only a few examples of items on which a user can act on the social networking system 730, and many others s are possible. A user can interact with anything that is capable of being represented in the social networking system 730 or in the external system 720, separate from the social networking system 730, or coupled with the social networking system 730 through the network 750.
[0135] The social networking system 730 is also capable of linking several entities. For example, the social networking system 730 allows users to interact with each other, as well as with external systems 720, or other entities, through an API, a network service or other communication channels. The system of
Petition 870160074035, of 12/08/2016, p. 44/66
40/54 social networks 730 generates and maintains the social graph, which comprises several nodes interconnected by various borders. Each node in the social graph can represent an entity, which can act on another node, and / or which can act through another node. The social graph can include several types of nodes. Examples of node types include users, entities other than people, content items, network pages, groups, activities, messages, concepts, and any other things that can be represented by an object in the 730 social networking system. An edge between two nodes in the social graph can represent a particular type of connection, or association, between two nodes, which can result from node relationships or an action that was performed by one of the nodes on the other node. In some cases, the edges between the nodes can be heavy. The weight of an edge can represent an attribute associated with the edge, such as an intensity of the connection or association between the nodes. Different types of edges can be supplied with different weights. For example, a border created when a user likes another user can be given a weight, while a border created when one acts as a friend to another user can be given a different weight.
[0136] As an example, when a first user identifies a second user as a friend, a border in the social graph is generated connecting a node, representing the first user, and a second node, representing the second user. As the various nodes refer to or interact with each other, the social networking system 730 modifies the edges connecting the various nodes to reflect the relationships and interactions.
[0137] The 730 social media system also includes user-generated content, which improves user interactions with the 730 social media system. User-generated content can include anything a user can add, transfer, send or post to the 730 social media system. For example, a user communicates posts to the 730 social media system from
Petition 870160074035, of 12/08/2016, p. 45/66
41/54 a 710 user device. Posts may include data, such as status updates or other text data, location information, images such as photos, videos, links, music or other similar data and / or media. Content can also be added to the 730 social media system by a third party. Content items are represented as objects on the social networking system 730. In this way, users of the social networking system 730 are encouraged to communicate with each other by posting text and content items from various types of media through various communication channels. This communication increases the interaction of users with each other and increases the frequency with which users interact with the 730 social networking system.
[0138] The social networking system 730 includes a network server 732, an API request server 734, a user profile store 736, a communication store 738, an action recorder 740, an activity record 744, and a digital video stabilization module 746. In one embodiment of the invention, the social networking system 730 can include additional components, less components or different components for various applications. Other components, such as network interfaces, security mechanisms, load balancers, switching standby servers, and network and control operations consoles, and the like, are not shown so as not to obscure the details of the system.
[0139] User profile storage 736 maintains information about user accounts, including biographical, demographic and other types of descriptive information, such as work experience, educational history, hobbies or preferences, location and the like, that have been declared by users or inferred by the social networking system 730. This information is stored in the user profile storage 736, so that each user is uniquely identified. The social networking system 730 also
Petition 870160074035, of 12/08/2016, p. 46/66
42/54 stores data describing one or more connections between different users in the 738 connection store. Connection information can indicate users who have similar or common work experience, group participants, hobbies or educational history. In addition, the 730 social networking system includes user-defined connections between different users, allowing users to specify their relationships with other users. For example, user-defined connections allow users to generate relationships with other users, which correspond to the real-life relationships of users, such as friends, coworkers, partners, and so on. Users can select from predefined connection types, or define their own connection types, if necessary. Connections to other nodes in the 730 social networking system, such as non-personal entities, deposits, group centers, images, interests, pages, external systems, concepts and the like are also stored in the 738 connection store.
[0140] The social networking system 730 maintains data on objects with which a user can interact. To maintain this data, user profile storage 736 and connection storage 738 store cases of the corresponding type of objects maintained by the social networking system 730. Each type of object has information fields, which are suitable for storing appropriate information for the type of object. For example, the user profile store 736 contains data structures with appropriate fields to describe a user account and information related to a user account. When a new object of a particular type is created, the social networking system 730 starts a new data structure of the corresponding type, assigns it a unique object identifier, and begins adding data to the object, if necessary. This can occur, for example, when a user becomes a user of the social networking system 730, the social networking system 730 generates a new case of a profile of
Petition 870160074035, of 12/08/2016, p. 47/66
43/54 user in user profile store 736, assigns a unique identifier to the user account, and begins to fill the user account fields with information provided by the user.
[0141] Connection storage 738 includes data structures suitable for describing user connections to other users, connections to external systems 720 or connections to other entities. The 738 connection store can also associate a connection type with user connections, which can be used in conjunction with the user privacy setting, to control access to user information. In one embodiment of the invention, user profile storage 736 and connection storage 738 can be implemented as an associated database.
[0142] The data stored in the connection store 738, the user profile store 736 and the activity log 742 allow the social network system 730 to generate the social graph, which uses nodes to identify the various objects and borders connecting the us, to identify the relationships between different objects. For example, if a first user establishes a connection with a second user on the social network system 730, the user accounts of the first user and the second sulfur of the user profile storage 736 can act as nodes in the social graph. The connection between the first user and the second user, stored by connection store 738, is an edge between the nodes associated with the first user and the second user. Continuing this example, the second user can then send a message to the first user within the social network system 730. The action of sending the message, which can be stored, is another border between the two nodes in the social graph, representing the first user and the second user. In addition, the message itself can be identified and included in the social graph as another node connected to the nodes representing the first user and the second user.
Petition 870160074035, of 12/08/2016, p. 48/66
44/54 [0143] In another example, a first user may indicate a second user in an image, which is maintained by the social networking system 730 (or, alternatively, in an image maintained by another system outside the social networking system 730 ). The image itself can be represented as a node in the 730 social networking system. This referral action can create borders between the first user and the second user, as well as create a border between each user and the image, which is also a node in the social graph. In yet another example, if a user confirms the attendance to an event, the user and the event are nodes obtained from user profile storage 736, where the event attendance is an edge between the nodes that can be retrieved from the registry of activities 742. By generating and maintaining the social graph, the social networking system 730 includes data describing many different types of objects and the interactions and connections between these objects, providing a rich source of socially relevant information.
[0144] The network server 732 connects the social networking system 730 to one or more user devices 710 and / or one or more external sources 720 through the network 750. The network server 732 serves network pages as well as well as other network-related content, such as Java, JavaScript, Flash, XML and so on. The network server 732 can include a mail server or other messaging functionality to receive and route messages between the social networking system 730 and one or more devices of 710 users. Messages can be instant messages, queued messages ( for example, email), text and SMS messages, or any other suitable messaging format.
[0145] The API 734 request server allows one or more external systems 720 and user devices 710 to request access information from the social networking system 730 by requesting one or more API functions. The API 734 request server can also allow external systems 720 to send information
Petition 870160074035, of 12/08/2016, p. 49/66
45/54 calls to the social networking system 730 requesting the APIs. The external system 720, in one embodiment, sends an API request to the social networking system 730, via the 750 network, and the API 734 request server receives the API request. The API 734 request server processes the request by requesting an API associated with the API request, to generate an appropriate response, which the API 734 request server communicates to the external system 720 over the 750 network. For example, responsive to the API request, the order server API 734 collects data associated with a user, such as user connections, which were recorded on external system 720, and communicates the collected data to external system 720. In another embodiment, user device 710 communicates with the social networking system 730 through APIS, in the same way as with the external systems 720.
[0146] The stock recorder 740 is able to receive communications from the network server 732 about user actions on and / or outside the social networking system 730. The stock recorder 740 fills the activity record 742 with information about the actions of users, allowing the social networking system 730 to discover various actions taken by its users within the social networking system 730 and outside the social networking system 730. Any action that a particular user takes with respect to another node in the social networking system social 730 can be associated with each user account, through information kept in the activity log 742 or in a database or other similar data warehouse. Examples of actions taken by a user within the 730 social networking system, which are identified and stored, may include, for example, adding a connection to another user, sending a message to another user, reading a message. another user's message, viewing content associated with another user, attending an event posted by another user, posting an image, attempting to post an image or other interaction actions with another user or other object. When a user takes action within the social media system
Petition 870160074035, of 12/08/2016, p. 50/66
46/54
730, the action is recorded in the activity log 742. In one embodiment, the social networking system 730 maintains the activity log 742 as a database of entries. When an action is taken within the social networking system 730, an entry for the action is added to the activity log 742. The activity log 742 can be referred to as an action log.
[0147] Additionally, user actions can be associated with concepts and actions that occur within an entity outside the social networking system 730, such as an external system 720, which is separate from the social networking system 730. For example , the stock recorder 740 can receive data describing a user interaction with an external system 720 from the network server 732. In this example, the external system 720 records a user interaction, according to the actions and objects structured in the social graph .
[0148] Other examples of actions in which a user interacts with an external system 720 include a user expressing an interest in an external system 720 or another entity, a user posting a comment on the social networking system 730, which discusses an external system 720 or a network page 722a within the external system 720, a user posting a Uniform Resource Locator (URL) on the social network system 730, or another identifier associated with an external system 720, a user attending an event associated with an external system 720, or any other action taken by a user, that is related to an external system 720. Thus, the activity log 742 can include actions describing interactions between a user of the social network system 730 and an external system 720, which is separate from the social networking system 730.
[0149] Authorization server 744 imposes one or more privacy settings on users of the 730 social networking system. A user's privacy setting determines how private information, associated with a user, can be shared. The privacy adjustment comprises the specification of
Petition 870160074035, of 12/08/2016, p. 51/66
47/54 private information associated with a user and the specification of the entity or entities with which the information can be shared. Examples of entities, with which information can be shared, may include other users, applications, external systems 720, or any entity that can potentially access the information. Information that can be shared by a user comprises user account information, such as profile pictures, phone numbers associated with the user, user connections, actions taken by the user such as adding a connection, changing user profile information, and similar.
[0150] The privacy adjustment specification can be provided at different levels of granularity. For example, the privacy setting can identify specific information that will be shared with other users; the privacy setting identifies a business phone number or a set of related information, such as personal information including profile picture, home phone number and status. Alternatively, the privacy adjustment may apply to all information associated with the user. The specification of the set of entities, which can access the particular information, can also be specified at various levels of granularity. Various sets of entities, with which information can be shared, may include, for example, all user friends, all friends of friends, all applications, or all external systems 720. One embodiment allows the specification of the set of entities comprises a list of entities. For example, the user can provide a list of external systems 720, which are allowed to have access to certain information. Another embodiment allows the specification to comprise a set of entities, together with exceptions, that do not allow access to information. For example, a user can allow all external 720 systems to access business information from the
Petition 870160074035, of 12/08/2016, p. 52/66
48/54 users, but specify a list of external 720 systems, which are not able to access business information. Certain embodiments ask for the list of exceptions, which cannot access certain information, a block list. External systems 720 belonging to a blacklist, specified by a user, are prevented from accessing the information specified in the privacy setting. Various combinations of information specification granularity and entity specification granularity, with which the information is shared, are possible. For example, all personal information can be shared with friends, while all business information can be shared with friends of friends.
[0151] The authorization server 744 contains logic to determine if certain information, associated with a user, can be accessed by friends of the user, external systems 720 and / or other applications and entities. External system 720 may need authorization from authorization server 744, so that it has access to the user's most private and sensitive information, such as the user's business phone. Based on the user's privacy settings, authorization server 744 determines whether another user, external system 720, an application, or another entity is allowed to access information associated with the user, including information about actions taken by the user.
[0152] The social networking system 730 may include a digital video stabilization module 746. The digital video stabilization module 746 can compute a set of refined camera orientations, subject to the restriction that empty regions are not visible, or below a minimum threshold value. The 746 digital video stabilization module can then generate distorted frames based on the set of new, refined camera orientations. In one embodiment, the digital video stabilization module 746 can be implemented as the digital video stabilization 400 of Figure 4.
Petition 870160074035, of 12/08/2016, p. 53/66
49/54
HARDWARE IMPLEMENTATION [0153] The processes and features presented above can be implemented across a wide range of machine and computer system architectures, and across a wide range of physical networking and computing media. Figure 8 illustrates an example of a computer system 800, which can be used to implement one or more of the embodiments described in this specification, according to an embodiment of the invention. The computer system 800 includes sets of instructions to make the computer system 800 perform the processes and characteristics discussed in this specification. The computer system 800 can be connected (for example, on a network) to other machines. In a networked arrangement, the computer system 800 can operate at the capacity of a server machine or a client machine on a client-server network medium, or as a paired machine on a point-to-point network medium ( or distributed). In one embodiment of the invention, the computer system 800 can be a component of the social networking system described in the present specification. In one embodiment of the invention, the computer system 800 can be a server among the many that make up all or part of the social networking system 830.
[0154] The computer system 800 includes an 802 processor, an 804 cache and one or more modules and executable units, stored in a computer-readable medium, directed to the processes and characteristics described in this specification. Additionally, the computer system 800 includes a high performance input / output (I / O) bus 806 and a standardized I / O bus 808. A host bridge 810 couples the 802 processor to the high performance I / O bus 806, while that the I / O bus bridge 812 couples the two buses 806 and 808 together. A system memory 814 and one or more 816 network interfaces are coupled to a high performance 806 I / O bus.
Petition 870160074035, of 12/08/2016, p. 54/66
50/54 computer system 800 may further include a video memory and a display device coupled to the video memory (not shown). Mass storage 818 and I / O ports 820 are coupled to the standardized I / O bus 808. The computer system 800 can optionally include a keyboard and pointing device, a display device, or other input / output devices (not shown) coupled to the standardized 808 I / O bus. Collectively, these elements are intended to represent a wide range of computer hardware systems, including, but not limited to, computer systems based on x86-compatible processors manufactured by Intel Corporation of Santa Clara, California, and x86-compatible processors manufactured by Advanced Micro Devices (AMD), Inc. of Sunnyvale, California, as well as any other suitable processor.
[0155] An operating system manages and controls the operation of the computer system 800, including the input and output of data in and from software applications (not shown). The operating system provides an interface between the software applications, which are running on the system, and the hardware components of the system. Any suitable operating system can be used, such as the LINUX Operating System, the Apple Macintosh Operating System available from Apple Computer Inc. of Cupertino, Calif., UNIX operating systems, Microsoft® Windows® operating systems, BSD operating systems and similar. Other implementations are possible.
[0156] The elements of the 800 computer system are described in more detail below. In particular, the network interface 816 provides communication between the computer system 800 and any of a wide range of networks, such as an Ethernet network (for example, IEEE 802.3), a chassis, etc. Mass storage 818 provides permanent storage for data and programming instructions to execute the processes and characteristics
Petition 870160074035, of 12/08/2016, p. 55/66
51/54 written above by the respective computing systems identified above, while system memory 814 (eg DRAM) provides temporary storage for data and programming instructions when executed by the 802 processor. The I / O 820 ports they can be one or more communication ports in series and / or in parallel, which provide communication between the additional peripheral devices, which can be coupled to the computer system 800.
[0157] Computer system 800 can include several system architectures, and various components of computer system 800 can be redeployed. For example, cache 804 can be a processor with 802 integrated circuit. Alternatively, cache 804 and processor 802 can be packaged together as a processor module, with processor 802 being referred to as the processor core. Furthermore, certain embodiments of the invention may neither require nor include all of the components mentioned above. For example, peripheral devices attached to the standardized 808 I / O bus can be attached to the high performance I / O bus. In addition, in some embodiments, only a single bus can exist, with computer system components 800 being coupled to the single bus. In addition, computer system 800 may include other components, such as other processors, storage devices or memories.
[0158] In general, the processes and characteristics described in this specification can be implemented as part of an operating system or application, a component, a program, an object, a specific module, or a series of specific instructions referred to as Software. For example, one or more programs can be used to perform the specific processes described in this specification. The programs typically comprise one or more instructions in various memories and storage devices in the computer system 800, which, when read and executed by one or more processes
Petition 870160074035, of 12/08/2016, p. 56/66
52/54 users, make the computer system 800 conduct operations to execute the processes and characteristics described in this specification. The processes and characteristics described in this specification can be implemented in software, hardware programming, hardware (for example, an application-specific integrated circuit), or any combination thereof.
[0159] In an implementation, the processes and characteristics described in this specification are implemented as a series of executable modules processed by the computer system 800, individually or collectively, in a distributed physical medium. The modules mentioned above can be executed by hardware, executable modules stored in a computer-readable medium (or machine-readable medium), or a combination of both. For example, modules can comprise several or a series of instructions, which will be executed by a processor in a hardware system, such as the 802 processor. Initially, the series of instructions can be stored on a storage device, such as mass storage 818. However, the instruction series can be stored in any suitable computer-readable storage medium. Furthermore, the instruction series does not need to be stored locally and can be received from a remote storage device, such as a server on a network, via the 816 network interface. The instructions are copied from the storage device, such as such as mass storage 818, in system memory 814 and then accessed and executed by the 802 processor. In several implementations, one or more modules can be executed by one processor or multiple processors in one or several locations, such as multiple servers in one physical means of processing in parallel.
[0160] Examples of computer-readable media include, but are not
Petition 870160074035, of 12/08/2016, p. 57/66
53/54 limited to, recordable type media, such as volatile and non-volatile memory devices; solid state memories; flexible and other removable disks; hard disk drives; magnetic media; optical discs (for example, Compact Read Only Memory Disc - CD ROMS, Digital Versatile Discs - DVDs); another tangible (or non-tangible), non-transitory (or transitory) storage medium; or any type of medium suitable for storing, coding or executing a series of instructions for execution by the computer system 800, to execute any one or more of the processes and characteristics described in this specification.
[0161] For explanatory purposes, several specific details are shown to provide a complete understanding of the description. As will be evident, however, to a person skilled in the art, embodiments of the invention can be practiced without these specific details. In some cases, modules, structures, processes, characteristics and devices are shown in the form of a block diagram, to avoid obscuring the description. In other cases, block diagrams and functional flowcharts are shown to represent data and logic flows. The components of the block diagrams and flowcharts (for example, modules, blocks, structures, devices, characteristics, etc.) can be combined in several ways, separated, removed, reordered and replaced in a different way than expressly described and illustrated in the this specification.
[0162] The reference in this specification to one embodiment, other embodiments, a series of embodiments, some embodiments, several embodiments or the like means that an aspect, project, structure or feature described together with the embodiment is included in at least least one embodiment of the invention. The appearances of, for example, the term in one embodiment, in various places in the specification are not ne
Petition 870160074035, of 12/08/2016, p. 58/66
54/54 all of them referring to the same embodiment, nor are they separate embodiments or alternatives mutually exclusive from other embodiments. Furthermore, whether or not there is an express reference to an embodiment or the like, several characteristics are described, which can be combined in various ways and included in some embodiments, but also omitted in various ways in other embodiments. Similarly, several characteristics are described, which may be preferences or requirements for some embodiments, but not other embodiments.
[0163] The language used in this specification was selected primarily for reading and instructional purposes, and was not selected to outline or circumscribe the inventive object. It is intended, therefore, that the scope of the invention is limited not by that detailed description, but, instead, by any of the claims that generate an application based on them. Consequently, the description of the embodiments of the invention is intended to be illustrative, but not limiting, of the scope of the invention, which is presented in the claims presented below.

权利要求:
Claims (20)
[1]
1. Process implemented by computer, CHARACTERIZED by the fact that it comprises:
providing, by means of a computer system, digital video data comprising a set of image frames having associated time periods and a set of camera orientation data having associated time periods, in which the set of camera orientation data it is based on a camera orientation when capturing digital video data by the camera; and generate, through the computer system, a refined set of camera orientation data by minimizing a rotation rate between successive image frames, while minimizing a number of empty regions in a refined set of resulting image frames reoriented with based on the refined set of camera orientation data.
[2]
2. Process according to claim 1, CHARACTERIZED by the fact that the amount of empty regions in the refined set of resulting image frames is minimized to zero.
[3]
3. Process according to claim 1, CHARACTERIZED by the fact that the amount of empty regions in the resulting refined set of image frames is minimized below a threshold value.
[4]
4. Process according to claim 1, CHARACTERIZED by the fact that it still comprises distorting the set of image frames based on the associated time periods for the set of image frames and the refined set of camera orientation data, for form a set of corrected image frames.
[5]
5. Process according to claim 4, CHARACTERIZED by the fact that it distorts the set of image frames based on time periods
Petition 870160074035, of 12/08/2016, p. 60/66
2/4 associated for the set of image frames and the refined set of camera orientation data, to form a set of corrected image frames, comprises:
dividing an individual image frame into a plurality of subsections, each subsection having an associated time frame and camera orientation; and realigning each subsection based on the associated time frame and camera orientation, to form a corrected individual image frame.
[6]
6. Process according to claim 4, CHARACTERIZED for still comprising displaying the set of corrected image frames as a video.
[7]
7. Process according to claim 6, CHARACTERIZED by the fact that the amount of empty regions in the refined set of resulting image frames is minimized below a threshold value.
[8]
8. Process according to claim 7, CHARACTERIZED in that it also comprises reconstructing the number of empty regions below a threshold value.
[9]
9. Process according to claim 1, CHARACTERIZED by the fact that the camera orientation data set having associated time periods, is provided from a gyroscope of a portable device.
[10]
10. Process according to claim 1, CHARACTERIZED by the fact that the set of picture frames is provided from a camera of a portable device.
[11]
11. Process according to claim 1, CHARACTERIZED by the fact that the camera orientation data set having associated time periods, is provided from a gyroscope of a portable device, comprising a mobile phone and a digital camera, and where the set of picture frames is provided from the digital camera.
Petition 870160074035, of 12/08/2016, p. 61/66
3/4
[12]
12. Process according to claim 1, CHARACTERIZED by the fact that generating the refined set of camera orientation data comprises an iterative optimization based on gradient decrease.
[13]
13. Process according to claim 1, CHARACTERIZED by the fact that generating the refined set of camera orientation data comprises filtering based on a Gaussian filter.
[14]
14. Process according to claim 1, CHARACTERIZED by the fact that generating the refined set of camera orientation data comprises filtering based on a time derivative.
[15]
15. Process according to claim 1, CHARACTERIZED by the fact that the camera orientation data set includes rotations without any translations.
[16]
16. Process according to claim 1, CHARACTERIZED by the fact that the camera orientation data set includes vectors having both rotations and translations.
[17]
17. Process according to claim 1, CHARACTERIZED by the fact that generating the refined set of camera orientation data is done by a social network system.
[18]
18. Process according to claim 17, CHARACTERIZED by the fact that the set of image frames having associated time periods and the camera orientation data set, having associated time periods, are uploaded to the social networking system by a user of the social networking system.
[19]
19. System, CHARACTERIZED by the fact that it comprises:
at least one processor, and a memory storing instructions configured to instruct at least one processor to perform:
Petition 870160074035, of 12/08/2016, p. 62/66
4/4 providing digital video data comprising a set of image frames having associated time periods and a set of camera orientation data having associated time periods; and generate a refined set of camera orientation data by minimizing a rotation rate between successive image frames, while minimizing an amount of empty regions in a refined set of reoriented image frames based on the refined set of image data. camera orientation, in which minimizing the rotation rate between successive image frames is based on at least a portion of the camera orientation data set.
[20]
20. Non-transitory computer storage medium storing executable instructions by computer that, when executed, causes a computer system to perform a computer-implemented process that comprises:
providing, by means of a computer system, digital video data comprising a set of image frames having associated time periods and a set of camera orientation data having time periods, where the camera orientation data set is based on a camera orientation when capturing digital video data by the camera; and generate a refined set of camera orientation data by minimizing a rotation rate between successive image frames, while minimizing an amount of empty regions in a refined set of reoriented image frames based on the refined set of image data. camera orientation, in which minimizing the rotation rate between successive image frames is based on at least a portion of the camera orientation data set.

类似技术:

---

公开号 | 公开日 | 专利标题

---

JP6605640B2|2019-11-13|System and method for stabilizing digital video via constraint-based rotational smoothing

---

JP6412129B2|2018-10-24|System and method for converting images

---

US10652465B2|2020-05-12|Systems and methods for combining multiple frames to produce media content with simulated exposure effects

---

US10725637B2|2020-07-28|Systems and methods for providing image perspective adjustment and automatic fitting

---

US20160148648A1|2016-05-26|Systems and methods for improving stabilization in time-lapse media content

同族专利:

---

公开号 | 公开日

---

US20140160309A1|2014-06-12|

---

CA2893802A1|2014-06-19|

---

US9071756B2|2015-06-30|

---

JP2016501497A|2016-01-18|

---

US9674440B2|2017-06-06|

---

KR20150095782A|2015-08-21|

---

MX345710B|2017-02-13|

---

US9554045B2|2017-01-24|

---

AU2017200248A1|2017-02-02|

---

KR102003012B1|2019-07-24|

---

AU2013359372B2|2016-10-13|

---

JP6608700B2|2019-11-20|

---

AU2018201829B2|2019-06-13|

---

JP6605640B2|2019-11-13|

---

IL239249A|2019-08-29|

---

CN104982027B|2018-07-27|

---

IL239249D0|2015-07-30|

---

EP2744192A2|2014-06-18|

---

EP2744192B1|2021-05-19|

---

AU2017200248B2|2017-12-14|

---

JP2018085775A|2018-05-31|

---

JP6937809B2|2021-09-22|

---

US20150002686A1|2015-01-01|

---

US20150222818A1|2015-08-06|

---

US20170180647A1|2017-06-22|

---

AU2013359372A1|2015-07-02|

---

JP2020005326A|2020-01-09|

---

AU2018201829A1|2018-04-12|

---

MX2015007499A|2015-12-15|

---

WO2014093517A1|2014-06-19|

---

US10511773B2|2019-12-17|

---

EP2744192A3|2016-09-14|

---

CA2893802C|2019-08-13|

---

CN104982027A|2015-10-14|

引用文献:

---

公开号 | 申请日 | 公开日 | 申请人 | 专利标题

---

---

AUPO793897A0|1997-07-15|1997-08-07|Silverbrook Research Pty Ltd|Image processing method and apparatus |

---

US6552744B2|1997-09-26|2003-04-22|Roxio, Inc.|Virtual reality camera|

---

US7015954B1|1999-08-09|2006-03-21|Fuji Xerox Co., Ltd.|Automatic video system using multiple cameras|

---

US7119837B2|2002-06-28|2006-10-10|Microsoft Corporation|Video processing system and method for automatic enhancement of digital video|

---

US7495694B2|2004-07-28|2009-02-24|Microsoft Corp.|Omni-directional camera with calibration and up look angle improvements|

---

US7542034B2|2004-09-23|2009-06-02|Conversion Works, Inc.|System and method for processing video images|

---

JP4793120B2|2006-06-21|2011-10-12|ソニー株式会社|Camera shake correction method, camera shake correction method program, recording medium storing camera shake correction method program, and camera shake correction apparatus|

---

US20080165280A1|2007-01-05|2008-07-10|Deever Aaron T|Digital video stabilization with manual control|

---

US7817187B2|2007-06-27|2010-10-19|Aptina Imaging Corporation|Image blur correction using a secondary camera|

---

US8207964B1|2008-02-22|2012-06-26|Meadow William D|Methods and apparatus for generating three-dimensional image data models|

---

JP5251410B2|2008-10-03|2013-07-31|株式会社ニコン|Camera work calculation program, imaging apparatus, and camera work calculation method|

---

US8363128B2|2009-08-10|2013-01-29|Samsung Electronics Co., Ltd.|Image processing apparatus and image processing method|

---

US20110085728A1|2009-10-08|2011-04-14|Yuli Gao|Detecting near duplicate images|

---

JP2011114649A|2009-11-27|2011-06-09|Sanyo Electric Co Ltd|Imaging device|

---

CN101917602B|2010-08-26|2012-07-04|四川大学|Digital video intelligent monitoring equipment based on high-resolution camera and data processing method|

---

US8488010B2|2010-09-21|2013-07-16|Hewlett-Packard Development Company, L.P.|Generating a stabilized video sequence based on motion sensor data|

---

US8531535B2|2010-10-28|2013-09-10|Google Inc.|Methods and systems for processing a video for stabilization and retargeting|

---

US8964041B2|2011-04-07|2015-02-24|Fr Vision Ab|System and method for video stabilization of rolling shutter cameras|

---

JP5734082B2|2011-05-11|2015-06-10|キヤノン株式会社|Imaging apparatus, control method therefor, and program|

---

US8823813B2|2011-06-06|2014-09-02|Apple Inc.|Correcting rolling shutter using image stabilization|

---

US20130107066A1|2011-10-27|2013-05-02|Qualcomm Incorporated|Sensor aided video stabilization|

---

US20130121559A1|2011-11-16|2013-05-16|Sharp Laboratories Of America, Inc.|Mobile device with three dimensional augmented reality|

---

US9041819B2|2011-11-17|2015-05-26|Apple Inc.|Method for stabilizing a digital video|

---

US20130162518A1|2011-12-23|2013-06-27|Meghan Jennifer Athavale|Interactive Video System|US20140184615A1|2012-12-28|2014-07-03|Nokia Corporation|Sequential Rendering For Field-Sequential Color Displays|

---

US9554048B2|2013-09-26|2017-01-24|Apple Inc.|In-stream rolling shutter compensation|

---

CN104954631B|2014-03-25|2018-02-27|腾讯科技（深圳）有限公司|A kind of method for processing video frequency, device and system|

---

US9413963B2|2014-05-30|2016-08-09|Apple Inc.|Video image stabilization|

---

US9596411B2|2014-08-25|2017-03-14|Apple Inc.|Combined optical and electronic image stabilization|

---

US9674438B2|2014-07-06|2017-06-06|Apple Inc.|Low light video image stabilization strength modulation|

---

CN105306804B|2014-07-31|2018-08-21|北京展讯高科通信技术有限公司|Intelligent terminal and its video image stabilization method and device|

---

US9398217B2|2014-10-15|2016-07-19|Microsoft Technology Licensing, Llc|Video stabilization using padded margin pixels|

---

US9392174B2|2014-12-11|2016-07-12|Facebook, Inc.|Systems and methods for time-lapse selection subsequent to capturing media content|

---

US9948920B2|2015-02-27|2018-04-17|Qualcomm Incorporated|Systems and methods for error correction in structured light|

---

US10068338B2|2015-03-12|2018-09-04|Qualcomm Incorporated|Active sensing spatial resolution improvement through multiple receivers and code reuse|

---

US9530215B2|2015-03-20|2016-12-27|Qualcomm Incorporated|Systems and methods for enhanced depth map retrieval for moving objects using active sensing technology|

---

US9635339B2|2015-08-14|2017-04-25|Qualcomm Incorporated|Memory-efficient coded light error correction|

---

US9846943B2|2015-08-31|2017-12-19|Qualcomm Incorporated|Code domain power control for structured light|

---

WO2017087537A1|2015-11-16|2017-05-26|Google Inc.|Stabilization based on accelerometer data|

---

CN105282400B|2015-11-20|2018-07-13|北京理工大学|A kind of efficient video antihunt means based on geometry interpolation|

---

WO2017107075A1|2015-12-22|2017-06-29|SZ DJI Technology Co., Ltd.|System, method, and mobile platform for supporting bracketing imaging|

---

EP3417606B1|2016-02-19|2020-08-12|FotoNation Limited|A method of stabilizing a sequence of images|

---

US9787902B1|2016-06-10|2017-10-10|Apple Inc.|Video image stabilization with enforced stabilization constraints|

---

AU2017344761A1|2016-10-20|2019-05-30|Spookfish Innovations Pty Ltd|An image synthesis system|

---

US10547784B2|2017-06-19|2020-01-28|SighTour Technologies, Inc.|Image stabilization|

---

US10477064B2|2017-08-21|2019-11-12|Gopro, Inc.|Image stitching with electronic rolling shutter correction|

---

WO2019054611A1|2017-09-14|2019-03-21|삼성전자 주식회사|Electronic device and operation method therefor|

---

US10462370B2|2017-10-03|2019-10-29|Google Llc|Video stabilization|

---

US10171738B1|2018-05-04|2019-01-01|Google Llc|Stabilizing video to reduce camera and face movement|

---

WO2019222059A1|2018-05-16|2019-11-21|Gopro, Inc.|Systems and methods for providing rotational motion correction|

---

WO2019222544A1|2018-05-16|2019-11-21|Iooi|System and method for navigating content from multiple users on a digital device|

---

US10587807B2|2018-05-18|2020-03-10|Gopro, Inc.|Systems and methods for stabilizing videos|

---

JP2020031264A|2018-08-20|2020-02-27|ソニーセミコンダクタソリューションズ株式会社|Signal processor, imaging apparatus, signal processing method|

---

US10750092B2|2018-09-19|2020-08-18|Gopro, Inc.|Systems and methods for stabilizing videos|

---

KR20200040541A|2018-10-10|2020-04-20|삼성전자주식회사|Camera module for stabilizing video, electrical video including the same and video stabilizing method of the electrical device|

---

US10911680B2|2018-12-28|2021-02-02|Intel Corporation|Method and system of geolocation and attitude correction for mobile rolling shutter cameras|

---

WO2020220292A1|2019-04-30|2020-11-05|深圳市大疆创新科技有限公司|Electronic image stabilization method, image acquisition device and movable platform|

---

WO2020257583A1|2019-06-21|2020-12-24|Gopro, Inc.|Systems and methods for stabilizing videos|

---

KR102176273B1|2019-07-04|2020-11-09|재단법인대구경북과학기술원|Method, system and computer program for video upright adjustment|

---

US11138804B2|2019-08-02|2021-10-05|Fmr Llc|Intelligent smoothing of 3D alternative reality applications for secondary 2D viewing|

---

CN110430365B|2019-08-26|2021-10-01|Oppo广东移动通信有限公司|Anti-shake method, anti-shake device, computer equipment and storage medium|

---

US11190689B1|2020-07-29|2021-11-30|Google Llc|Multi-camera video stabilization|

法律状态:
2018-11-21| B06F| Objections, documents and/or translations needed after an examination request according [chapter 6.6 patent gazette]|

---

2020-07-21| B15K| Others concerning applications: alteration of classification|Free format text: AS CLASSIFICACOES ANTERIORES ERAM: H04N 5/225 , H04N 5/262 Ipc: H04N 5/232 (2006.01), H04N 5/225 (2006.01), H04N 5 |

---

2020-07-28| B06U| Preliminary requirement: requests with searches performed by other patent offices: procedure suspended [chapter 6.21 patent gazette]|

---

2020-11-24| B11B| Dismissal acc. art. 36, par 1 of ipl - no reply within 90 days to fullfil the necessary requirements|

---

2021-10-13| B350| Update of information on the portal [chapter 15.35 patent gazette]|

优先权:

---

申请号 | 申请日 | 专利标题

---

US201261735976P| true| 2012-12-11|2012-12-11|

---

US14/101,252|US9071756B2|2012-12-11|2013-12-09|Systems and methods for digital video stabilization via constraint-based rotation smoothing|

---

EP13196486.8A|EP2744192B1|2012-12-11|2013-12-10|Systems and methods for digital video stabilization via constraint-based rotation smoothing|

---

PCT/US2013/074444|WO2014093517A1|2012-12-11|2013-12-11|Systems and methods for digital video stabilization via constraint-based rotation smoothing|

---